Skip to main content

Uscensus Connection

The goal of this document​

Here, we are going to connect the vendor called US census to the Airbyte then it will redirect the data using the destination of MySQL, User can choose the format of the destination as required, For example from MySQL/ PostgreSQL, etc.

Useful Resources of Airbyte​

Requirements to get the ‘US census’ source​

Steps to Fetch the Personal API Token from ‘US census’​

  1. Go to the https://airbyte.com/ and open the Airbyte source setup window.

  2. Select the US census from the source type, it will show the required properties to set up the source:

Docusaurus Slash Introduction

  1. Once the required properties appear, click on the link beside the API key title to reach to the page where you can get the API key of Uscensus. or go to the https://api.census.gov/data/key_signup.html

Docusaurus Slash Introduction

  1. Add required details to Request the key, Add details like Organization Name, Email Address! After adding the details Click on the Submit key request button to get the API key.

Docusaurus Slash Introduction

  1. After clicking the Submit button, you will get the confirmation of API key delivery over your email, as shown in the figure below:

Docusaurus Slash Introduction

  1. The API key will be delivered on the Email which you've added in the details, as shown in the Figure below:

Docusaurus Slash Introduction

Steps to Connect UScensus with AIV​

  1. Go to the Airbyte using the below link: https://airbyte.com/

  2. The Airbyte landing screen looks like below:

Docusaurus Slash Introduction

  1. Here We have 3 options on the left, which contain Connections, Sources, and Destination. As per continuing the steps click on the Sources option.

Docusaurus Slash Introduction

  1. To add the UScensus source, click on the New Source button from the Top-right corner.

Docusaurus Slash Introduction

  1. To add Any of the sources here, the user needs to add the properties which are needed from the Airbyte, every source has its own individual properties.

Docusaurus Slash Introduction

note

Note: Here every source demands a different type of property, where Client DI, Client Secret, and Refresh Tokens are common in most sources. on that we can say, all the sources are different from each other.

  1. Set up source Form overview:
  • Name: Users can add the name of the source as per their requirements.
  • Source Type: The user has to select the source type from the provided list by airbyte.
  • API key: The API key provide the connection between the vendor and users application, it helps to get the public data from the vendor's end.
  • query_path: Add the query path from the above given examples.
  • query_params: -

Docusaurus Slash Introduction

  1. Now, set up the source here,
  • Add the name as US census,
  • Select uscensus from the source,
  • Add API key in the form,
  • Add Query path from the above example as data/2019/cbp
  • Validate your form with as shown in the figure below:

Docusaurus Slash Introduction

  • Now, click on Set up Connection button, it will test the connection here, then it will redirect to add destination of the source, as per figure below:

Docusaurus Slash Introduction

  • Alert example of the connection successfully in figure below:

Docusaurus Slash Introduction

  1. After testing the connection successfully add the destination here.

Docusaurus Slash Introduction

Here, on the source adding screen,

  • The top-menu bar shows two buttons where 1. for Overview of the source and 2. for the settings of the source.
  • Overview: The overview screen shows the details related to the source; it may be empty as per the above screen.
  • Settings: From settings, the user can edit the form details of the added source.
  • Add destination button leads us to the destination page from this source page.
  1. Click on the Destination from the Top-right corner, as shown in the figure below:
note

It’s will show the available destinations, here user can add the new destination as per required.

Docusaurus Slash Introduction

info

To know more about adding the required destination in Airbyte go to the Destination

  1. Click on the MySQL Training here. it may take some time to fetch the stream names, as shown in figure below:

Docusaurus Slash Introduction

  1. After loading all the Streams form the Vendor’s to the destination, the destination page looks as shown in the figure below:

Docusaurus Slash Introduction

  1. After loading all the data streams user need to add Sync frequency and Table Prifix, as shown in the figure below:
  • Add Sync Frequency: Manual
  • Add Table Prefix: aiv_
  • As shown in the figure below:

Docusaurus Slash Introduction

• When the user add the prefix, it gets added on Destination stream name automatically, as shown in the figure below:

Docusaurus Slash Introduction

  1. At the bottom of the destination page, we have two radio buttons, which contain the Normalization option. Keep the Basic Normalization selected.

Docusaurus Slash Introduction

  1. Now, click on the setup connection to complete the connection. it will test the connection again which may take some time.

Docusaurus Slash Introduction

  1. Now, to validate the data is synced by the Airbyte, Go to the Connection from the Left menu bar of the screen:

Docusaurus Slash Introduction

  1. Find the Added connection of the UScensus from the list,

Docusaurus Slash Introduction

Here, the Data isn’t synced yet, for that it shows the icon of non-sync at the start of the row. click on the row to see the details.

  1. Here the UScensus connection page will show the Status of the data sync and sync history.

Docusaurus Slash Introduction

  1. The title of the source and destination name.
  2. Status and settings page top menu bar.
  3. Enable button: from this button, the user can enable/disable the source from the destination.
  4. Reset your data and Sync button: the user can reset their data and only sync the updated data from the vendor.
  1. Click on the sync button to sync the data manually, after clicking on the sync button it will start the process which will indicate the status under the history grid.
info

Once it gets completed it will indicate as the Succeeded under the history, as it will also indicate the size of data records number and time of sync. Here we have added the sync time as every hour, on that the Airbyte will sync the data in every hour automatically which will be shown as the figure below:

Docusaurus Slash Introduction